feat: harness contract and plugin SPI (plan 1 of 3) by uipreliga · Pull Request #181 · UiPath/coder_eval

uipreliga · 2026-09-16T18:04:57Z

Summary

Plan 1 of 3 of the harness target architecture: every agent declares what it honors of the shared agent config, and a task that asks for something the chosen harness cannot do is rejected at resolution, so coder-eval plan fails before any run.

HarnessContract per agent class. Each agent declares which of system_prompt, plugins, permission_mode, allowed_tools and disallowed_tools it honors, the permission_mode values it honors, and whether it supports cooperative stop. Registration fails without a contract. The ClassVars supports_cooperative_stop, supports_cost_log_tags and system_prompt_semantics are gone.
A closed tool vocabulary. CANONICAL_TOOL_NAMES (14 names, checked against the pinned Claude Code CLI) and a per-harness ToolNameMap that must map every name. Names are checked at resolution, with a did-you-mean hint.
Tool restrictions honored natively. Pi: --tools / --no-tools / --exclude-tools. OpenCode: permission rules and an instructions file for the system prompt. Antigravity: SDK tool-call policies. permission_mode: plan means read-only on all three. The Codex tool forward did nothing (enabled_tools is an MCP-server key) and is deleted.
Per-kind defaults. An experiment's defaults.agent.by_type.<kind> applies only when the final agent kind matches, below the task. Claude-only defaults in experiments/default.yaml moved there.
Resolution-time check. validate_harness_contract raises HarnessContractError, a TaskResolutionError. plan, run and export stop on it. Every "set but NOT enforced" warning is deleted.
Plugin surface. coder_eval.spi with SPI_VERSION = 1. cost_log_tags is a base-constructor kwarg on Agent.
Guardrails. Lint rule CE068 keeps concrete agent kinds out of orchestration/, streaming/ and timing.py. make parity-table generates the contract and tool tables in docs/agents/HARNESS_PARITY.md, and CE069 fails on drift. tests/test_harness_conformance.py requires one offline probe per enforced cell and per declared permission mode, and checks that a system prompt never rides the user turn.
Bugs fixed from review. Pi and OpenCode admitted one turn over max_turns. Antigravity re-pulled the step stream on any RuntimeError, and swallowed a failed teardown silently. The early-stop ceiling could divide by zero.

Breaking changes: YAML that loads today is now rejected

A Codex task that sets permission_mode, allowed_tools or disallowed_tools.
permission_mode: acceptEdits or default on Pi, OpenCode or Antigravity (they honor plan and bypassPermissions). A task-level acceptEdits run with --type pi is rejected too; put it under by_type.claude-code.
A tool name outside the canonical set. Claude Code still accepts mcp__… names and rules such as Bash(git status:*).
plugins, system_prompt, permission_mode or a tool list on a type: none task.
experiments/default.yaml now applies model, permission_mode and allowed_tools to Claude Code only, so --type pi no longer inherits the Claude model id.
An empty allowed_tools: [] restricts nothing on every harness, as on Claude Code.

Before merge

Delegate (coder_eval_uipath): its agent fails registration until it declares a contract and accepts cost_log_tags. Land that change, and move permission_mode under by_type.claude-code in the RPA pipeline's shared experiment YAML, together with or before this PR.
One live run per harness with allowed_tools: [Read]: Pi and OpenCode (Haiku 4.5 via OpenRouter) and Antigravity (gemini-3.5-flash; it runs Gemini only) each wrote no file and scored 0, while an unrestricted control run on the same model passed on all three. An extra Antigravity run with only disallowed_tools: [Bash] scored 1.0 with no Bash call, so its policies are selective.

Open follow-ups (simulator safeguard, the acceptEdits owner question, deferred lint ideas) are tracked in c/harness-target-architecture-v2.md and .claude/harness-candidates.md.

Test plan

make verify: 5989 passed, 2 skipped, 93% coverage; ruff, pyright, custom lint (CE000+) and docs budget clean
coder-eval plan over every task: exit 0; plan -e over every shipped experiment: exit 0 except the intended with-plugin rejection on the agentless task
mkdocs build --strict; new anchors verified in the built HTML
Docker staging and Harbor export round-trip tests: a reloaded non-Claude task passes the contract check
Live smoke runs on Pi, OpenCode and Antigravity: restricted runs wrote nothing, unrestricted controls passed (see Before merge)

🤖 Generated with Claude Code

…date registrations Replace the supports_cooperative_stop and system_prompt_semantics ClassVars with one frozen HarnessContract per agent class. AgentRegistry.register now rejects an agent without a contract, and a config class that is not a forbid-extra BaseAgentConfig whose type Literal names the kind. The base environment info records the contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ructor Agent.__init__ is now concrete and takes cost_log_tags; every in-tree agent forwards it through super().__init__. The orchestrator forwards the tags on every LiteLLM route, so the supports_cost_log_tags capability gate is deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…and Antigravity Pi maps allowed_tools / disallowed_tools / permission_mode plan to --tools, --no-tools and --exclude-tools. OpenCode writes explicit permission rules and an instructions file for system_prompt into OPENCODE_CONFIG_CONTENT and always passes --auto. Antigravity builds SDK tool-call policies. Every map is the inverse of the adapter's telemetry map. The Codex enabled_tools / disabled_tools forward was a no-op (they are MCP-server keys) and is deleted with its task. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…on-time harness contract check An experiment's defaults.agent may carry by_type.<kind>, applied below the task and selected by the final agent kind, so Claude-only defaults move out of the shared baseline. A gated agent field set on a harness whose contract marks it unsupported now raises HarnessContractError, which plan, run and export treat as a hard config error. The sdk_options guard is derived from the registry, a -D system_prompt_file is inlined after layer 5, and every unenforced-field warning is deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…map per harness HarnessContract gains permission_modes, the permission_mode values a harness honors with their Claude Code meaning. ToolNameMap maps every name in CANONICAL_TOOL_NAMES to the harness's native tools (an empty tuple for a tool the harness lacks), is validated at registration, and replaces the silent .get(name, ()) reads. Pi and OpenCode smoke tasks move to bypassPermissions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…mes a harness cannot honor The resolution-time contract check now also rejects a permission_mode value outside the harness's permission_modes and a tool-list name outside CANONICAL_TOOL_NAMES, with a did-you-mean hint. Claude Code keeps its native mcp__ names and permission rule syntax. Orchestrator setup runs the check for library and in-container runs, but not for a re-grade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…oder_eval.spi surface CE068 flags a concrete agent config class or an AgentKind member (other than UNKNOWN) named in orchestration/, streaming/ or timing.py. coder_eval.spi re-exports the stable plugin surface with SPI_VERSION = 1, and EXTENDING.md documents the harness contract a plugin agent must declare. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…the harness conformance test make parity-table renders the HarnessContract and ToolNameMap of every in-tree agent into docs/agents/HARNESS_PARITY.md, and CE069 fails on drift. The conformance test derives every rejection from the contracts and requires one offline probe per enforced cell and declared permission mode, including a check that a system prompt never rides the user turn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

- Docker staging and the Harbor export write only the agent fields a layer set, so a reloaded non-Claude task no longer claims the permission_mode default and fails the contract check in the container (round-trip tests added). - An empty allowed_tools / disallowed_tools restricts nothing and is not a set field. - OpenCode keeps a host rule for external_directory / doom_loop under an allowlist. - coder_eval.spi exports CommandTelemetry, TokenUsage, TranscriptMessage, ResultSummary and ModelPricing, which a plugin needs to emit events and prices. - Correct stale notes and docs: the Codex start() warning, the simulator's allowed_tools=[] claim, plan meaning, resolution vs load, and Pi's ls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…eardown and the early-stop ceiling - Pi and OpenCode stop at the (N+1)th turn_start / step_start before it is counted or emitted, so max_turns=N records N turns (it recorded N+1). - Antigravity retries a receive_steps() RuntimeError only when no step was pulled yet; a later error is a real failure and is no longer re-pulled. - A failed Antigravity harness teardown is logged instead of swallowed silently. - The early-stop ceiling fails closed at zero armed weight instead of dividing by zero, matching armed_criteria_passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

+subagent tool.
+"""
+
+TOOL_NAME_ALIASES: Final[dict[str, str]] = {"Task": "Agent"}


… task caps Co-Authored-By: Claude <noreply@anthropic.com>

uipreliga · 2026-09-16T21:59:03Z

Stacked on this PR: #183 (plan 2 of 3, central enforcement) targets feat/harness-contract. Merge this one first; #183 then retargets to main.

CarlesUIPath · 2026-09-18T15:41:09Z

Nice PR, the joint contract for cross-harness consistency improves eval reliability. I have ran a multi-model review and a few smoke tests on my end. Below are some findings:

OpenCode: a host's non-tool `allow` is silently flipped to `deny` by rule ordering (Medium)

What coder-eval emits (deterministic, from _inject_config_content):

host {external_directory: allow} + task allowed_tools: [Read, Bash]
  -> emitted: {external_directory: allow, "*": deny, doom_loop: allow, read: allow, bash: allow}
  -> the kept host allow sits at index 0, BEFORE "*": deny at index 1

The kept host rule lands before our "*": deny because we drop our own copy of it
(opencode_agent.py:1018-1019), while the design intent — the "OpenCode applies the LAST
matching rule" comment at :1014 and OPENCODE.md:197 — assumes our rules come last. For a
kept non-tool key they don't. The host-deny direction is safe only coincidentally (both
entries are deny); the ordering is wrong in both cases but only bites when the host wants
allow.

Confirmed live (opencode via Azure Codex gpt-5.6-luna). Read of a file outside the
project root via the read tool (bash denied), varying only permission order:

A  external_directory: deny  (last)                          -> BLOCKED  (confirms the lever)
B  external_directory: allow (last)                          -> ALLOWED  (allow wins when last)
C  external_directory: allow BEFORE "*": deny  <-- emitted   -> BLOCKED

B and C differ only in ordering; C is the exact config coder-eval produces for a host that set
external_directory: allow plus a task allowlist, and it resolves to DENY. A denied
external_directory blocks out-of-sandbox access — where the system-prompt file and skills
under test live — so the host's (and our own default's) intended allow is silently lost.

This matches the two semantics the adapter's own code already assumes: last-match-wins, and
re-allowing _NON_TOOL_PERMISSIONS after permission["*"] = "deny" in _permission_config
(:964-965) — only necessary if "*" matches those non-tool keys, which the live result
confirms it does.

Fix (verified at the emit layer): insert kept inherited non-tool rules after our entries —
yields allow/deny correctly in both directions. Worth a unit test pinning the emitted rule
order. Trigger is narrow (host pre-sets `OPEnon-tool allow AND the
task sets an allowlist), hence Medium.

CodeQL check (currently red) — 3 errors + 1 note

3× py/missing-call-to-init (error), test-only. Agent.__init__ is now concrete
(takes cost_log_tags, sets lifecycle sta __init__ without
super().__init__(...) are flagged: tests/fixtures/mock_agent.py:15 (__init__ at :30),
tests/fixtures/text_stub_agent.py:17 (_ tests/test_simulation_integration.py:106 (_ExplodingAgent.initat :111). Add thesuper().init(...) call to each (theyontract = stub_contract();
the init call was missed).
**1× py/unused-global-variable (note) — ALIASES (models/enums.py:148) is used in models/harness_contract.py(import +from_inverse`
alias expansion); the query only reasons wfe to dismiss.

Low / informational (no change required,

**disallowed_tools: ["Task"] on claude-cium; verification
brought it to Low. The claude-code adapter forwards disallowed_tools raw
(claude_code_agent.py:1179, :1199) andIASESitself (identity map at:709), unlike the from_inverse` harnesses — so it relies on the CLI resolving the
alias. Tested the CLI directly via the modtool list:
```
control                 -> subagent ad
--disallowedTools Agent -> `Task` removed
--disallowedTools Task  -> `Task` remo
```
The pinned CLI resolves the alias in both . Residual concern is
only that this rests on undocumented CLI behavior the adapter neither owns nor tests — a
conformance probe asserting both spellingsch a future regression.
(CLI 2.1.197; PR pins 2.1.216.)
OpenCode's edit permission key couples Wirections
(documented CLI coarseness).
Simulator _SIMULATOR_DISALLOWED_TOOLS liharmless given the
result above, but list both spellings for clarity.
command_executed matches tool_name witle judge_context /
summaries accept both spellings — inconsistent; normalize at the one matching seam.
Pi plan + an allowlist collapses to --nocumented divergence from claude-code plan`.
plan validates merge layers 1–4 only (skexport/harbor still
catch a -D violation, so not a correctness hole.

uipreliga and others added 11 commits September 16, 2026 07:50

chore: defer three harness candidates from the harness contract plan

3039db6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-advanced-security Bot found potential problems Sep 16, 2026

View reviewed changes

Comment thread src/coder_eval/models/enums.py

subagent tool.

"""

TOOL_NAME_ALIASES: Final[dict[str, str]] = {"Task": "Agent"}

fix(limits): clarify docstring for RunLimits class to better describe…

474638f

… task caps Co-Authored-By: Claude <noreply@anthropic.com>

uipreliga mentioned this pull request Sep 16, 2026

feat: central enforcement — TurnMonitor, live budgets, plugin staging (plan 2 of 3) #183

Open

uipreliga mentioned this pull request Sep 17, 2026

feat: TurnEmitter and the ports — one turn kernel on every harness (plan 3 of 3) #184

Open

uipreliga marked this pull request as ready for review September 18, 2026 00:31

uipreliga requested review from CarlesUIPath, akshaylive, bai-uipath and tmatup as code owners September 18, 2026 00:31

Navigation Menu

feat: harness contract and plugin SPI (plan 1 of 3) - #181

feat: harness contract and plugin SPI (plan 1 of 3)#181
uipreliga wants to merge 12 commits into
mainfrom
feat/harness-contract

uipreliga commented Sep 16, 2026 •

edited

Loading

Uh oh!

uipreliga commented Sep 16, 2026

Uh oh!

CarlesUIPath commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

uipreliga commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes: YAML that loads today is now rejected

Before merge

Test plan

Uh oh!

uipreliga commented Sep 16, 2026

Uh oh!

CarlesUIPath commented Sep 18, 2026

OpenCode: a host's non-tool allow is silently flipped to deny by rule ordering (Medium)

CodeQL check (currently red) — 3 errors + 1 note

Low / informational (no change required,

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

uipreliga commented Sep 16, 2026 •

edited

Loading

OpenCode: a host's non-tool `allow` is silently flipped to `deny` by rule ordering (Medium)